I Want to Believe: Journalists and Crowdsourced Accuracy Assessments in Twitter

Evaluating information accuracy in social media is an increasingly important and well-studied area, but limited research has compared journalist-sourced accuracy assessments to their crowdsourced counterparts. Œis paper demonstrates the di‚erences between these two populations by comparing the features used to predict accuracy assessments in two TwiŠer data sets: CREDBANK and PHEME. While our €ndings are consistent with existing results on feature importance, we develop models that outperform past research. We also show limited overlap exists between the features used by journalists and crowdsourced assessors, and the resulting models poorly predict each other but produce statistically correlated results. Œis correlation suggests crowdsourced workers are assessing a di‚erent aspect of these stories than their journalist counterparts, but these two aspects are linked in a signi€cant way. Œese di‚erences may be explained by contrasting factual with perceived accuracy as assessed by expert journalists and non-experts respectively. Following this outcome, we also show preliminary results that models trained from crowdsourced workers outperform journalist-trained models in identifying highly shared “fake news” stories.